Although the Crusaders reached the head of the Gulf of Aqaba in 1116, they did not take control of the oasis of Aila, which was in Muslim hands as late as 1154. In the early 1160s, the Crusaders built a castle on Pharaoh's Island. The name île de Graye, by which the island and castle are commonly known, is a 19th-century invention from Arabic qurayya, "little village". Contemporary chronicles call it Aila, like the oasis.

In December 1170, Saladin conquered the island and reconstructed the citadel and left a garrison of men there. In November 1181, Raynald of Châtillon raided the Arab-held Aila and attempted to set up a naval blockade against the Muslim troops there during the winter of 1182 to 1183. The blockade consisted of only two ships and was not successful. By the time of the 13th century, when the pilgrim Thietmar passed the island in 1217, the entire place was inhabited by a fishing village and populated by Muslims and captive. The Mamluk governor of the city of Aqaba lived in the citadel until some time in the 14th century, around 1320, when the seat of governorship was moved into the city itself.

Modern Pharaoh's Island

Along with the citadel of El-Gendi Fortress, also on the Sinai Peninsula about halfway between Nekhel and Suez, the citadel on Pharaoh's Island was added onto the UNESCO World Heritage Tentative List on July 28, 2003, due to its purported universal cultural value.

Because of its location near Jordan and Israel, the island and its coral reefs have become a popular sightseeing attraction among tourists based in Taba, Eilat, and Aqaba.